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Abstract 

(ST 

Memoryless channels with deletion errors as defined by a stochastic channel matrix allowing for bit 
(N. drop outs are considered in which transmitted bits are either independently deleted with probability d or 

q | unchanged with probability 1 — a. Such channels are information stable, hence their Shannon capacity 

exists. However, computation of the channel capacity is formidable, and only some upper and lower 
bounds on the capacity exist. In this paper, we first show a simple result that the parallel concatenation 
of two different independent deletion channels with deletion probabilities d\ and d 2 , in which every input 
bit is either transmitted over the first channel with probability of A or over the second one with probability 
of 1 — A, is nothing but another deletion channel with deletion probability of d = \d\ + (1 — X)d2- We then 
provide an upper bound on the concatenated deletion channel capacity C{d) in terms of the weighted 



(N 



CZ2 



■ average of C(di), C{d2) and the parameters of the three channels. An interesting consequence of this 

_ bound is that C(\d\ + (1 — A)) < AC(e?i) which enables us to provide an improved upper bound on 



(N 



the capacity of the i.i.d. deletion channels, i.e., C{d) < 0.4143(1 — d) for d > 0.65. This generalizes 
the asymptotic result by Dalai Q] as it remains valid for all d > 0.65. Using the same approach we are 
£N) . also able to improve upon existing upper bounds on the capacity of the deletion/substitution channel. 

Index Terms 

X 

Deletion channel, deletion/substitution channel, channel capacity, capacity upper bounds. 

I. Introduction 

Channels with synchronization errors can be well modeled using bit drop outs and/or bit insertions as 
well as random errors. There are many different models adopted in the literature to describe these errors. 
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Among them, a relatively general model is employed by Dobrushin (H where memoryless channels with 
synchronization errors are described by a channel matrix allowing for the channel outputs to be of different 
lengths for different uses of the channel. As proved in the same paper, for such channels, information 
stability holds and Shannon capacity exists. However, the determination of the capacity remains elusive 
as the mutual information term to be maximized does not admit a single letter or finite letter form. 

In the existing literature, several specific instances of this model are more widely studied. For instance, 
by a proper selection of the stochastic channel transition matrix, one obtains the i.i.d. deletion channel 
which represents one of the simplest models allowing for bit drop-outs which is the model considered in 
this paper. In a binary i.i.d. deletion channel, the transmitted bits are either received correctly and in the 
right order or deleted from the transmitted sequence altogether with a certain probability d independent 
of each other. Neither the receiver nor the transmitter knows the positions of the deleted bits. Despite the 
simplicity of the model, the capacity for this channel is still unknown, and only a few upper and lower 
bounds are available El-O. Other special cases of the general model by Dobrushin are the Gallager 
model allowing for insertions, deletions and substitution errors in which every transmitted bit is either 
deleted with probability of d, replaced with two random bits with probability of i, flipped with probability 
of / or received correctly with probability of 1 — d — i — /. Substituting i = in the Gallager model 
results into the deletion/substitution channel model which is also considered in this paper. Another look 
at the deletion/substitution channel can be as a series concatenation of two independent channels such 
that the first one is a deletion only channel with deletion probability of d and the second one is binary 
symmetric channel (BSC) with cross error probability of s = y^-j. There are also some capacity upper 
and lower bounds for the Gallager's deletion channel model in the literature, e.g., lP71-ll9l. 

In this paper, we prove that the capacity of an i.i.d. deletion channel with deletion probability of d as an 
arithmetic mean of two different deletion probabilities d\ and di, d = Xdi + (1 — X)d2 for A 6 [0, 1], 
can be upper bounded in terms of the capacity and the parameters of the two newly considered deletion 
channels. The proof relies on the simple observation that the deletion channel with deletion probability 
d can be considered as the parallel concatenation of two independent deletion channels with deletion 
probabilities d\ and d<i where each bit is either transmitted over the first channel with probability A or 
the second channel with probability 1 — A. 

Thanks to the presented inequality relation among the deletion channels capacity, we are able to 
improve upon the existing upper bounds on the capacity of the deletion channel for d > 0.65 O. 
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The improvement is the result of the fact that the currently known best upper bounds are not convex 
for some range of deletion probabilities. More precisely, our result allows us to convexify the existing 
deletion channel capacity upper bound for d > 0.65, leading to a significant improvement of the upper 
bound. In other words, we are able to prove that for < A < 1, C(\d + 1 — A) < \C(d), resulting 
in C{d) < 0.4143(1 — d) for d > 0.65 which is tighter than the result in (6]]. The same result for the 
asymptotic scenario d — > 1 was also obtained in flTJ using a different approach; however our result is 
valid for d > 0.65 hence more general. We also note that the best known limiting lower bound (as 
d — > 1) is 0.1185(1 — d) Q. We also demonstrate that a similar improvement is possible for the case 
of deletion/substitution channels. As an example, we can prove that for s = 0.03, an improved capacity 
upper bound is obtained for d > 0.6 over the best existing result given in 0. 

The paper is organized as follows. In Section [TTJ we prove the main result of the paper which 
relates the capacity of the three different deletion channels through an inequality. In Section [Till we 
generalize the result to the case of deletion/substitution channels and the parallel concatenation of more 
than two channels. In Section Jvj we present tighter upper bounds on the capacity of the deletion and 
deletion/substitution channels based on previously known best upper bounds, and comment on the limit 
of the capacity as the deletion probability approaches unity. We conclude the paper in Section [V] 

II. Main Theorem 

In this section, we provide the main result of the paper on the capacity of the deletion channel and its 
proof. Furthermore, we present a simple proof for the special case with d 2 = 0, i.e., C(Xd± + 1 — A) < 
\C(dx). 

The theorem below states our basic result whose proof hinges on a simple observation. 

Theorem 1. Let C(d) denotes the capacity of the i.i.d. deletion channel with deletion probability d, 
A £ [0, 1] and d = Xd\ + (1 — X)d2, then we have 

C(d) < AC(di) + (l-A)C(d 2 ) + (l-d)log(l-d) 

-A(l - di) log(A(l - di)) - (1 - A)(l - da) log((l - A)(l - d 2 )). (1) 

Proof: Let us consider two different deletion channels, C\ and C 2 , with deletion probabilities d\ and 
d 2 , input sequences of bits X± and X 2 , and output sequences of bits Yi and Y2, respectively. Denote 
their Shannon capacities by C(d\) and C(d 2 ), respectively. Given a specific A 6 (0,1), define a new 
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Fig. 1. Channel Model C' 



binary input channel C (shown in Fig. [T]) with input sequence of bits X and output sequence of bits Y 
as follows: each channel input symbol is transmitted through C\ with probability A, and through C 2 with 
probability 1 — A, independently of each other. Neither the transmitter nor the receiver knows the specific 
realization of the "individual channel selection events," i.e., they do not know which specific subchannel 
a symbol is transmitted through, and which specific subchannel each output symbol is received from. The 
following two lemmas demonstrate that 1) the new channel is a new i.i.d. deletion channel with deletion 
probability d = Xd\ + (1 — A)(i 2 , 2) if appropriate side information be provided for the transmitter and 
the receiver then the capacity of the genie-aided channel is upper bounded by 

AC7(di) + (l-A)C(^) + (l-d)Iog(l-£0-A(l-di)bg(A(l-di))-(l-A)(l-d2)log((l-A)(l-d2)). 

Combining these two results, the proof of the theorem follows easily by noting that the capacity of the 
new channel C' cannot decrease with side information. ■ 
The following two lemmas are employed in the proof of the theorem. 

Lemma 1. C as defined in the proof of the theorem above is nothing but a deletion channel with deletion 
probability d = Xdi + (1 — A)c^2- 

Proof: For each use of the channel C , for any input symbol x £ X and channel output y € y, 
the transition probability is given by P{C\ is usedjdi + P{C2 is used}^ = Xd\ + (1 — X)d2- Noting 
that the subchannels are memoryless and the channel selection events are independent of each other, this 
transition matrix precisely defines a deletion channel with deletion probability d = Xd± + (1 — A)e?2. ■ 

Lemma 2. The capacity of the channel C as defined in the proof of the theorem above is upper bounded 
by 

A<7(d 1 ) + (l-A)a(d 2 ) + (l-d)log(l-d)-A(l-di)log(A(l-di))-(l-A)(l-d 2 )log((l-A)(l-d 2 )). 
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Proof: We first define a new genie-aided channel which is obtained by providing the transmitter 
and the receiver of the channel C with appropriate side information, then derive an upper bound on 
the capacity of the genie-aided channel which is also an upper bound on the capacity of the channel C. 
More precisely, we provide the transmitter with side information on which channel is being used for each 
transmitted symbol (X = X1X2), and the receiver with side information on which channel the received 
symbol comes from (Y = Y1Y2), and reveal the side information on the fragmentation information, i.e., 
random process F y , to the receiver such that by knowing F y , Y\ and Y2, one can retrieve 1^. F y is 
defined as an M-tuple F y = [f y [l], • • • , f y [M]), where M denotes the length of the received sequence 
Y, i.e., M = \Y\, and f y [i] S {1,2} denotes the index of the channel the i-th received bit is coming 
from. We also define F x which determines the fragmentation process from the random process X to X\ 
and X2 as an iV-tuple F x = (/^[l], • • • , f x [N]), where f x [i] G {1, 2} denotes the index of the channel 
the i-th bits is going through. 

Since X — > (X±,X 2 , F x ) — > {Y\, Y2, F y ) — > Y form a Markov chain, we can write 

I(X;Y) < I(X 1 ,X 2 ,F x ;Y 1 ,Y 2 ,F y ) 

= I 1+ I 2 + I 3 , (2) 

where I x = I(X U X 2 , F x] Y 1 ), I 2 = I(X 1 , X 2 , F X ;Y 2 \Y 1 ) and I 3 = I(X 1 , X 2 , F x ; F y \Y 1 ,Y 2 ). 
For I\, we have 

h = I(X 1 ;Y 1 ) + I(X 2 ,F X ;Y 1 \X 1 ) 

= nxr^), (3) 

where we used the fact that P(Yi\X\, X2, F x ) = P(Yi\X\), i.e., Y\ is independent of X2 and F x 
conditioned on X 1 . Furthermore, by using the facts that P(Y 2 \X 2 , Yi) = P{Y 2 \X2) and P{Y 2 \Xi, X 2 , F x , Yi) = 
P(1^2|^2), we obtain 

I 2 = I(X 2 ;Y 2 \Y 1 ) + I(X l ,F x ;Y 2 \Y 1 ,X 2 ) 
= H(Y 2 \Yi) — H{Y 2 \X 2 ) 

< I(X 2 ;Y 2 ). (4) 

We are not able to derive the exact value of ^3, therefore we derive an upper bound on ^3 which 
results in an upper bound on I(X,Y). For ^3, if we define Ni = \Xi\ and Mj = \Yi\ as the length of 



the transmitted and received sequences form the z-th channel, respectively, then we can write 
^3 = H(F y \Yi,Y 2 ) — H(F y \Yi,Y 2 , Xi, X 2 , F x ) 
< H(F y \Y 1 ,Y 2 ) 

= H(F y \M 1 ,M 2 ). (5) 

For fixed Mi and M 2 , there are ( Mi m^ 2 ) possibilities for F y = (/„[()],••■ ,/jMi]). Therefore, we 
obtain 

H(F y \M 1 = Mi,M 2 = M 2 ) < log (( Ml ^ M2 )) 

< (Mi + M 2 ) log (Mi + M 2 ) - Mi log(Mi) - M 2 log(M 2 ), (6) 

where we have used the inequality log (£) < nHf,^) provided in flTOl p. 353]. Due to the fact that 
(x + a)log(x + a) — xlog(x) is a concave function of x for a > 0, and E{Mi\M 2 = M 2 } = 
(N — M 2 ) x+(il d \)d 2 ( see Appendix lA)) . by applying Jensen's inequality, we can write 

J 3 < E MiM2 {H(F y \M 1 ,M 2 )} 

< £ Ma j(£{Mi|M 2 } + M 2 ) log(S{Mi|M 2 } + M 2 ) 

-£{Mi|M 2 }log(£{Mi|M 2 }) - M 2 log(M 2 ) 

Furthermore since (a(6 — x) + x) log(a(6 — x) + x) — a(6 — x) log(a(6 — x)) — xlog(x) is a concave 
function of x for a > and < x < b, and i?{Af 2 } = N(l — A)(l — d 2 ) (see Appendix lAb. by applying 
Jensen's inequality, we obtain 

h < A f (A(l-di) + (l-A)(l-d 2 ))log(iV(A(l- C ?i) + (l-A)(l-d 2 ))) 

-N\(l - dt) log(iVA(l - di)) - N(l - A)(l - d 2 ) log(iV(l - A)(l - d 2 )) 
= N{\{\ - dx) + (1 - A)(l - tfe)) log(A(l - di) + (1 - A)(l - d 2 )) 

-iVA(l - di) log(A(l - di)) - N{1 - A)(l - da) log((l - A)(l - d 2 )). (8) 



On the other hand, for I(Xf, Y"j) (i € {1, 2}), we can write 

I(Xi]Yi) = HXiiYuNj-HXiiNilYi) 

= I{Xi- Yi\Ni) + I(X l ; iVj) - I(X l ; N t \Yi) 

< I(X i ;Y i \N i ) + H(N, i ) 

< IiXi-YilN^ + logiN + l) 

N 

= ^2 p ( N i = Ni)I(Xi] Yi\Ni = Ni) + log(iV + 1), (9) 

Ni=0 

where in deriving the first inequality we have used the facts that H(Ni\Xi) = and I(Xf, Ni\Yi) > 0, 
and in deriving the second equality the fact that 

H(Ni) = -J2 (^) An (! " X ) N ' n lo § ( Q An (! " X ) N ~ n ) < 1 °S(N + 1). (10) 

Furthermore, as it is shown in 0, for a finite length transmission over the deletion channel, the mutual 
information rate between the transmitted and received sequences can be upper bounded in terms of the 
capacity of the channel after adding some appropriate term, which can be spelled out as flSJ Eqn. (39)] 

I(X i ;Y i \N i = N i )<N i C(di) + H(D i \N i = N i ), (11) 

where Di denotes the number of deletion through the transmission of 2Vj bits over the z-th channel and 

H{Di\Ni = Ni) = ~Y, ( n0^ (1 " d ^~ niog (( n0^ (1 " ~ l ° g{Nl + ^ 

Substituting (TTTb into ©, we have 

N 

/(X,; Yi) < Yl P ( N * = N i) ( N i C (di) + log(JVi + 1)) + \og(N + 1) 

Ni=0 

< A i iVC( ( i l ) + log(A l iV + l) + log(iV + l), (12) 

where the last inequality results since log(x) is a concave function of x, and Ai = A and A2 = 1 — A. 
Finally, by substituting (02]), ®> © and © in ©, we obtain 

I(X;Y) < NXC(d 1 ) + log(XN + l) + N(l-X)C(d 2 ) + log((l-X)N + l) 

+ 2 log(JV + 1) + N(l - d) log(l - d) - NX{1 - d{) log(A(l - d x )) 

-N(l - A)(l - d 2 ) log((l - A)(l - d 2 )). 

By dividing both sides of the above inequality by N, letting N go to infinity, and noting that the inequality 
is valid for any input distribution P(X), the proof follows. ■ 
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Note that for the special case of C2 being a pure deletion channel, i.e., d 2 = 1, the presented upper 
bound (fl"5l) results into C(Adi + 1 — A) < AC(di). One can observe that to prove the relation C(Adi + 
1 — A) < AC(di), there is no need for the entire proof given in Lemma |2 More precisely, when C 2 is a 
pure deletion channel, X — > X% — > Y\ — > Y form a Markov chain (Y = Y±), therefore we can write 

I(X;Y) KliX^Yj) 

< \NC(dx) + log(AiiV + 1) + log(iV + 1), (13) 

where the last inequality holds due to (fT2l ). Furthermore, by dividing both sides of the above inequality 
by N, letting N go to infinity, and the fact that the inequality is valid for any input distribution P(X), 
we arrive at C(Xdi + 1 - A) < AC(di). 

Another observation from the result C(\d\ + (1 — A)) < \C{d\) is that by series concatenation of 
two independent deletion channels with deletion probabilities d\ and 1 — A, we also arrive at a deletion 
channel with deletion probability of d = \d\ + 1 — A. Therefore we can say that the capacity of the series 
concatenation of two independent deletion channels can be upper bounded in terms of the capacity of 
one of them and the parameters of the other. 

III. Some Generalizations and Implications 

A. Generalization to the Case of Deletion/Substitution Channel 

In a deletion/substitution channel (special case of the Gallager channel model without any insertions) 
with parameters (d,f), any transmitted bit is either deleted with probability of d or flipped with probability 
of / or received correctly with probability of 1 — d— f, where neither the transmitter nor the receiver have 
any information about the position of the deleted and flipped bits. It is easy to show that the result of 
Theorem Q] can also be generalized to the deletion/substitution channel as given in the following corollary. 

Corollary 1. Let C(d, f) denotes the capacity of the deletion/substitution channel with deletion prob- 
ability d and flip probability f, A € [0, 1], d = Xd\ + (1 — A)c?2 and f = A/i + (1 — A)/2, then we 
have 

C(d,f) < AC7(d 1 ,/ 1 ) + (l-A)C7(d 2 ,/ 2 ) + (l-d)log(l-d) 

-A(l - di) Iog(A(l - di)) - (1 - A)(l - d 2 ) log((l - A)(l - da)). (14) 

Proof: The proof of Lemma Q] simply holds if we consider C\ in Fig. Q] as a deletion/substitution 
channel with parameters (di,/i) and C2 as another deletion/substitution channel with parameters (da,f2), 
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then C becomes also a deletion/substitution channel with parameters (Xd± + (1 — X)d,2, Xfx + (1 — A)/2). 
Furthermore, replacing the deletion channel Cj with deletion probability di with a deletion/substitution 
channel with parameters (di,fi) does not change the distribution of AT; and M*. Therefore, the proof of 
Lemma |2] holds for the deletion/substitution channel as well. ■ 
Note that a deletion/substitution channel with parameters (d, /) can be considered as a series concatena- 
tion of two independent channels where the first one is a deletion only channel with deletion probability 
of d and the second one is a binary symmetric channel (BSC) with cross error probability s = 
(1 — d — f < 1 and if d = 1 then s = 0). If we define C s (d, s) = C(d, (1 — d)s), then for efo = 1 and 
/2 = 0, we obtain 

C s (Xdx + l-X,s) <XC s (d!,s). (15) 

B. Parallel Concatenation of More Than Two Channels 

So far, we considered the parallel concatenation of two independent deletion channels which is useful 
in improving upon the existing upper bounds. However, we can also consider the parallel concatenation 
of more than two deletion channels. If we define the deletion channel C as a parallel concatenation of P 
independent deletion channels C p with deletion probability d p (p = {1, • • • , P}) where each input bit is 
transmitted with probability X p over C p , and modify the definition of F y such that f y [i] € {1, • • • ,P} 

denotes the index of the channel the z-th bit is coming from, then for d = J2p=i \d p , we have 

p p 
C(d) < \C{d p ) + {l-d) log(l - d) - Ap(l - d p ) log(A p (l - d p )), (16) 
p=i p=i 

where J2p=i \ = 1- Note, however, that this result does not give any tighter upper bounds on the 

deletion channel capacity than the one obtained by considering the parallel concatenation of only two 

independent deletion channels. 

IV. Improved Upper Bounds on the Deletion Channel Capacity 

An interesting application of the result £T|) on the capacity of the deletion and deletion/substitution 
channels is in obtaining improved capacity upper bounds. For instance, the best known upper bound 
on the deletion channel capacity is not convex for d > 0.65 as shown in Fig. |2] (with values taken 
from the boldfaced values in Table IV of [6]). As clarified in the table, the best known values for 
small d are due to ifTTTl . for a wide range (up to d ~ .8) are due to the "fourth version" of the upper 
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bound (named C4 in [6]), and for large values of d are due to the "second version" named C% in the 
same paper. Therefore, the deletion channel capacity upper bound can be improved for d G (0.65, 1) as 
C(l - 0.35A) < AC(0.65) < AC 4 (0.65) with < A < 1. That is, we have C(d) < 0.4143(1 - d) for 
d G (0.65, 1). This is illustrated in Fig. |3] 




Fig. 2. Previously best known upper bound on the i.i.d. deletion channel capacity. 




0.6 0.65 0.7 0.75 0.8 0.85 0.9 0.95 1 

d 



Fig. 3. Improved upper bound on the deletion channel capacity employing C(Xd + 1 — A) < \C(d). 



1 1 

We note that our result is a generalization of the one in [1J where it was shown that C(d) < 0.4143(1 — 
d) as d — > 1. We also note an earlier asymptotic result on a lower bound derived in Q which states that 
(7(d) as d ->• 1 is larger than 0.1185(1 - d). 



s= 0.03 




Fig. 4. Previously best known upper bound on the deletion/substitution channel capacity for s = 0.03. 
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Fig. 5. Improved upper bound on the deletion/substitution channel capacity for s = 0.03. 
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As another application of the inequality derived in this paper, we can consider the capacity of the 
deletion/substitution channel. The best known capacity upper bound for this case is given in Q, e.g., 
Fig. 1 of Q presents several upper bounds for fixed s = 0.03 (see Fig. H}. It is clear that this bound is 
not a convex function of the deletion probability for d > 0.6, hence it can be improved. That is, applying 
the result in our paper, we obtain, for instance for s = 0.03, C s (d, 0.03) < 0.3621(1 — d) for d > 0.6 
which is a tighter bound as illustrated in Fig. [5] 

V. Conclusions 

In this paper, an inequality relating the capacity of a deletion channel to two other deletion channels 
is found. The main idea is to consider parallel concatenation of two different independent deletion 
channels and relate the capacity of the resulting deletion channel with the capacity of the first two. An 
immediate application of this result is in obtaining improved upper bounds on the capacity of the deletion 
channel as the best available upper bounds are not convex in the deletion probability, and the derived 
inequality results in a tighter capacity characterization. For an i.i.d. deletion channel, we proved that 
C(d) > 0.4143(1 — d) for all d > 0.65. This is a stonger result than the earlier characterization in HI 
which is valid only asymptotically as d — > 1. We also noted a generalization of the result to the case of 
a deletion/substitution channel and provided a tigher capacity upper bound for this case as well. 
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Appendix A 
Stochastic Properties of M 1 and M 2 

For P(M\, M 2 ), we can write 

n-m 2 

P(M 1 = M 1 ,M 2 = M 2 ) = Yl P(Mi = M 1 ,M 2 = M 2 \N 1 = Ni)P(N 1 =Ni) 

N 1 =M 1 
N-M 2 

= P(Mi = M 1 \N 1 = Ni)P(M 2 = M 2 \Nx= Ni)P(N 1 = Ni) 

N 1= M-l 

(*)a*(i-a)"-* 

(A(l-di)) Ml ((l-A)(l-d 2 )) M2 x 
E 2 ( N " (M1 m M2) ) (Adi)^- A/l (d - A)*)™-"- 

,r. _ n.f. \ 1 1 / 

(A(l - di)) Ml ((l - A)(l - d 2 )) M *d N ~ Ml - M \ (17) 



JVi=Mi 



A r - M 2 \ / iV 
Mi J VM 2 

7V-M; 

X 

N 1= M 



N - M 2 \ ( N 
Mi ) \M 2 

Furthermore, due to the structure of the channel C', M 2 is binomially distributed, i.e., P(M 2 = M 2 ) = 
( A ^)((l - A)(l - d 2 )) hh (\ + (1 - A)^)^-^, and as a result E{M 2 } = N(l - A)(l - d 2 ). On the 
other hand, to obtain E ^ ^{M \\M 2 } , we first need to obtain P(Mi\M 2 ), for which we can write 

P(M X ,M 2 ) 



P{Mi = M X \M 2 = M 2 ) 



Therefore, we obtain 

N—M 2 



P(M 2 ] 
N — M 2 
Mi 



(A(l - d 1 )) Ah (Xd 1 + (1 - X)d 2 ) N ~ M ^ Ah (X + (1 - X)d 2 ) M2 ~ N . 



^ Mi {M!|M 2 } = £ Mi ( N M M2 ) (A(l - di)) Ml (Adi + (1 - X)d 2 ) N - M ^(X + (1 - A)*)*-" 

M 1= ^ 1 ' 

A(l-di) 
A + (1-A)d 2 ' 



(iV - M 2 ) ~ ^ . (18) 
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